Repository navigation
perf(telemetry)!: index trace logs per batch (batch format 6) - #507
Merged
Merged
Conversation
Every batch with logs gets logs.fidx: entries of trace hash, posting offset and count, then one u32 logs.parquet row per traced row. The index is a pure function of the logs file. Commit and compaction build it from the file they just wrote, in hash-range passes above 8M traced rows, and offline verification rebuilds it and compares every entry and posting.
Loading a batch keeps the first hash of every 256-entry block of trace.fidx and logs.fidx. A probe binary-searches the fences and reads one block with a single pread, instead of about 17 reads per lookup. The file is still opened per probe, so a catalog of thousands of batches holds no index descriptors.
ParquetStore.TraceLogs probes each batch's logs.fidx and reads only the posted rows, seeking to each run and decoding the projected columns. Rows keep the exact trace ID, window and namespace rule of the span read, and the first limit rows by (time, service, span ID, batch, row), which refines the DuckDB query's time order. Trace detail reads spans and logs from one pinned catalog capture through TraceDetail, and the MCP path admits that read as one interactive read, as it admitted the DuckDB log query.
Trace detail, trace logs and drill-down summaries reopened and reparsed each span or log file's footer and page indexes per read, which costs as much as the rows read from a large compacted file. A cache bounded by 64 MiB of encoded footer and page-index bytes keeps recently read files parsed and open. Retiring a batch drops its entries, and a held file closes when its last reader releases it. At 10M log rows compacted, the warm span read drops from 10.4 to 0.9 ms and the log read from 7.0 to 0.9 ms.
Format 6 adds logs.fidx to every batch with logs; format 5 batches are rejected like any other unsupported format, so upgrading servers start with empty telemetry. The Parquet physical schema is unchanged. The panel benchmark replay rebuilds log indexes for tiled batches, and AGENTS.md, the storage, data-layout and capabilities docs, and the diagram sources describe the new sidecar.
Random batches with empty trace IDs, one trace ID in two namespaces, logs-only batches and forced ties are read through logs.fidx and by a full scan in the same total order, after commit and after every compaction pass up to the second generation, across windows, namespace rules and limits. Every compacted batch then passes offline verification. A crafted hash collision keeps only the exact trace ID and fails verification.
The byte bound alone let many small uncompacted files hold hundreds of descriptors. The cache now also keeps at most 512 files open.
Building logs.fidx through a row reader assembled and converted every row and added about 105 ms to a 100k-row logs commit. Reading the trace_id chunk's pages directly brings logs commit latency back to the pre-index 65 ms.
…ex builds on cancellation Index sidecar loaders now report short content as a format error and return other read failures unchanged, so a failing device is never classified as a corrupt batch. The log index build checks its context before every page, so a preempted compaction stops promptly; verification passes its context too and cancellation is never treated as corruption. Closing the Parquet store closes the footer cache's idle files and leaves held ones to close on release.
The panel benchmark assembled logs.fidx with its own copy of the format. WriteLogIndex exposes the store's builder for tools that assemble batch directories, so the benchmark can no longer drift from the format.
Trace reads go through TraceDetail, which pins one snapshot for spans and logs. The trace benchmark now times that read instead of a path production never takes.
An exported writer for the bench is unreachable from production, which the deadcode check rejects. The bench keeps its writer, as it does for trace.fidx.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trace detail found a trace's logs with
trace_id = ?over every log file in the window. Logs sort by time bucket, namespace, service and time, so nothing pruned that filter and every lookup decoded thetrace_idof every retained log row. On a Hetzner ccx33 with about 1.9B rows retained, all 250 trace-detail timeouts failed in that query, and drill-down and annotations queued behind it.Each batch with logs now has
logs.fidx, which maps a trace hash to the trace's log rows. It is built fromlogs.parquetby one function used by commit, compaction and offline verification, which rebuilds the index and compares it. Trace detail reads the posted rows in Go through the Parquet offset index, under the same catalog snapshot as the spans, and keeps read admission. Trace index probes for both signals go through in-memory fences and read one block. Parsed footers of recently read trace files are cached, bounded, and dropped on retirement.Results are exact. Logs are ordered by
(log_time, service, span_id, batch, row), a total order that refines the oldORDER BY time. An oracle test compares against a full scan after commit and after compaction.Breaking: batch format 6. Format 5 data is rejected, so upgraded servers start with empty telemetry.
Laptop benchmark (
internal/observability/trace_logs_bench_test.go, 100M log rows):Commit latency for a 100k-row batch is unchanged within noise. Proposal:
docs/proposals/0013-log-trace-lookup.md.